User Guide

The MOP application in the Itential Automation Platform (IAP) allows customers to build and execute automated activities that provide the ability to run network health checks by collecting and evaluating operational data from the network.

Template Types

The MOP application supports two template types, Command and Analytics. Both template types share similar behaviors:

• Pass/Fail Evaluations
• Rule Severities
• Variable Processing

Command Templates allow for device command responses to be evaluated against a set of rules.

Analytics Templates allow for changes in device responses over time to be evaluated by comparing the results of two separate command template executions.

Prerequisites

1. Working NSO instance.
   • At least one device with a live-status capable NED. Netsim devices may not work.
   • itential_tools NSO package preloaded
2. An instance of IAP with the following installed:
   • MOP
   • Device Manager
   • Workflow Builder (WFB)
   • Workflow Engine (WFE)

Command Templates

Command Templates are run-time templates that actively pass commands to a list of specified devices during their runtime. After all responses are collected, the output set is evaluated against a set of defined rules. These executed templates are referred to as Pre and Post steps, which are typically separated by a procedure (router upgrade, service migration, etc).

Creating Command Templates

To create a new Command Template:

1. Click New Command Template.

   

2. The Command Template Designer page opens. During design, you can execute commands on physical devices to see the actual response and rules evaluation.

   

3. Enter a template name (required). Click Save.

   

4. Enter the device name in the search field (upper right) to select a device.

5. Add the relevant command variables and rules. For more detailed information, see the sections on Defining Commands and Command Variables below.

6. Click Test this Command. After evaluating the command output, rules will be marked with either a green check mark (passed) or red minus sign (failed). The device response will appear in the black box at the bottom of the screen.

   

7. Click Delete to remove a saved template. To clone a saved template, edit the saved template name and click Update.

   

Defining Commands

Command templates contain sets of commands with associated rules. Click Add Command to add a new command. Click Add Rule to add a new rule to a command.

Every command may have one or more rules which are bound by a boolean Flag called a "pass flag".

• When a pass flag is set to true (on), the logic between the rules is "AND". This means all rules must pass in order for the command to pass.
• When a pass flag is set to false (off), the logic is "OR". This means that only one rule has to pass for the command to pass.

A Severity field is available for each rule. The severity only applies to a rule after it has failed. The default severity is error.

The global pass flag (All/One commands must pass) is located above the Command list. This works the same as the individual "pass flags" on each command, except on a global level. There is also a pass checkbox (optional) to ignore failures due to warning or info severity levels. The default for the pass checkbox is "off'.

• All commands must pass (default behavior): Pass for every rule in every command.
• All commands must pass (warn/info as pass checked): Zero errors errors present in the template.
• One command must pass (default behavior): Pass for all rules in ONE (or more) commands.
• One command must pass (warn/info as pass checked): One command with zero errors must be present in the template.

To remove a command, click the delete icon (trashcan) to remove all of its associated rules.

Command Variables

When defining command templates, variables may be used in the commands with the following syntax:

<!VARIABLE!>

The following variable types can be used when defining commands:

• string
• number
• boolean
• array of strings
• array of numbers
• array of booleans

When arrays of simple types are passed into the execution engine, the command will be executed one time for each element in the array. Complex variable structures such as arrays of complex objects are not currently supported.

Rule Types

The following rule types are available in all command templates.

Rule Type	Description
contains	Verify the specified string exists in the device response.
!contains	Verify the specified string does not exist in the device response.
contains1	Verify the specified string exists only once in the device response.
RegEx	Verify the regular expression matches the device response.
!RegEx	Verify the regular expression does not match the device response.
#comparison	Compare two values, both of which may be extracted from the device response.

Flags

The following flags are supported.

Flag	Description
i	Ignore case. Make the whole expression case-insensitive.
g	Global search. Retain the index of the last match, allowing iterative searches.
m	Multi-line. Beginning/end anchors (^/$) will match the start/end of a line.

Defining Rules

The contains and RegEx operators search for command output matching or not matching a given string or regular expression. The #comparison operator evaluates the values in returned data.

Rule Type	Match Type to Device Response	Output	Supported Flag
contains	String	True if the device response contains the requested value.	Ignore case (i).
!contains	String	True if the device response does not contain the requested value.	Ignore case (i).
contains1	String	True if the device response contains 1 and only 1 occurence of the requested value.	Ignore case (i).
RegEx	Regular expression comparison	True if the device response matches the requested regex pattern.	Ignore case (i). Global search (g). Multi-line (m)
#comparison	Comparison against two values, both of which may be extracted from the device response	The #comparison rule type defines its own set of Operands (Top and Bottom) and a Comparison Operator. See below for more information.

Top Comparison Operand

Uses a regular expression with a capture group to select data in the output of a command. The regular expression must be surrounded with slash (/) characters.

The expression serves two functions: it identifies the command output location (where we will find the top comparison operand value) and returns the comparison operand value.

Capture groups are created by surrounding the relevant part of the regular expression with round brackets or parentheses. A single capture group is supported.

Bottom Comparison Operand

Will specify a static value, a passed MOP command template variable, or a second regular expression and capture group matching part of the command's output.

Comparison Operators

Operator	Comparison Type	Description
=	String	Returns true if the top and bottom string comparison operands are equal; otherwise returns false.
!=	String	Returns true if the top and bottom string comparison operands are not equal; otherwise returns false.
<=	Number	Returns true if the top number comparison operand is equal to or less than the bottom number comparison operand; otherwise returns false.
<	Number	Returns true if the top number comparison operand is less than the bottom number comparison operand; otherwise returns false.
>=	Number	Performs a number comparison. Returns true if the top number comparison operand is greater than or equal to the bottom number comparison operand; otherwise returns false.
>	Number	Returns true if the top number comparison operand is greater than the bottom number comparison operand; otherwise returns false.
%	Number	Evaluates a percentage value calculation against a threshold. The percentage value is calculated by multiplying 100 times the bottom number comparison operand divided by the top number comparison operand: 100 x (bottom comparison operand / top comparison operand) . The threshold is a static number typed in the Max accepted % field. Returns true if the percentage value is less than or equal to the threshold; otherwise it returns false.

Configuration Examples

Use the following examples to configure various rules and comparison operators and operands for a command template.

Example 1

In this example, we need to verify a Cisco CSR has sufficient memory. The IOS command show platform software vmemory info prints memory statistics, including the total system memory.

In the MOP application:

1. Create a new Command Template.
2. Select a device name for testing.
3. Type show platform software vmemory info in the command field of Command #1.
4. Keep the default contains operator.
5. Click Test This Command.
6. MOP displays the test results.

Example Test Results

Memory Upgrade Limits:
    Total System Memory:        3890 MB
    Memory From Upgrade Licenses:   0 MB
    Memory From Feature Licenses:   0 MB
Memory Available For Upgrade:
    Available System Memory:    0 MB
    Available Upgrade Licensed Memory:  0 MB
    Available Feature Licensed Memory:  0 MB
Current Memory Allocation:
    IOSD:   2354 MB (default) + 0 MB upgrade
    Data Plane: 1536 MB (default) + 0 MB upgrade

Top Comparison Operand

To build a regular expression to select data:

1. Select RegEx operator.
2. Type a regular expression to select the command's output data containing the first operand. The line containing "Total System Memory" includes the first operand, 3890 MB.
3. Type Total System Memory:\s+\d+ MB in the rule field.
4. Click Test This Command.
5. Verify a green check is displayed, indicating the rule matches.
6. If a red dash is displayed, check the regular expression for errors.

To build a capture group:

1. Change the command operator to #comparison.

2. Surround the regular expression rule with slash characters:

   /Total System Memory:\s+\d+ MB/

3. Add a capture group by placing a pair of parentheses around the part of our regular expression matching installed memory:

   /Total System Memory:\s+(\d+) MB/

To set the comparison operator:

1. Select >= comparison operator.

Bottom Comparison Operand

1. Type 3000 in the compare with field.
2. Click Test This Command.
3. Verify a green check is displayed, indicating the rule matches.
4. If a red dash is displayed, check the top comparison operand regular expression for errors.

Example 2

This example extends the previous example; we need to verify a Cisco CSR has sufficient memory. In the previous example, we compared installed memory to a static value. In this example, we compare installed memory to a variable.

Variable MEM-THRESH will be passed to the MOP command template.

Bottom Comparison Operand

• Type in the Compare with field.

Example 3

In this example, we have an EBGP adjacency configured with a maximum prefix of 10 prefixes. The adjacency is configured to log warning messages when the neighbor sends 80% or more of the maximum number of prefixes, and the adjacency is reset if the neighbor sends more than 10 prefixes. In this example, we verify the number of sent prefixes is not greater than the warning threshold.

The IOS command show ip bgp neighbor X.X.X.X prints the number of accepted prefixes, the maximum prefix limit, and the warning threshold.

1. In the MOP application, create a new command template.

2. Select a device name for testing.

3. Type show ip bgp neighbor X.X.X.X in the command field for Command 1.

   • Replace X.X.X.X with a valid EBGP neighbor's router ID.

4. Keep the default contains operator.

5. Click Test This Command.

6. MOP displays the test results.

   Router_B#show ip bgp neighbor 10.0.0.1
   BGP neighbor is 10.0.0.1,  remote AS 200, external link
   BGP version 4, remote router ID 10.0.0.1
   BGP state = Established, up for 00:13:22
   Last read 00:00:21, hold time is 180, keepalive interval is 60 seconds
   Neighbor capabilities:
       Route refresh: advertised and received(old & new)
       Address family IPv4 Unicast: advertised and received
       IPv4 MPLS Label capability:
   Received 930 messages, 0 notifications, 0 in queue
   Sent 919 messages, 1 notifications, 0 in queue
   Default minimum time between advertisement runs is 30 seconds
   
   For address family: IPv4 Unicast
   BGP table version 30, neighbor version 30
   Index 1, Offset 0, Mask 0x2
   Route refresh request: received 0, sent 0
   9 accepted prefixes consume 432 bytes
   Prefix advertised 0, suppressed 0, withdrawn 0, maximum limit 10 (warning-only
   )
   Threshold for warning message 80%
   
   Connections established 2; dropped 1
   Last reset 00:29:13, due to BGP Notification sent, update malformed
   Message received that caused BGP to send a Notification:
       FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
       003C0200 00001940 01010040 02040201
       00C84003 040A0000 01800404 00000000
       180A000A 180A000B 180A000C
   External BGP neighbor can be up to 2 hops away.
   Connection state is ESTAB, I/O status: 1, unread input bytes: 0
   Local host: 10.0.0.2, Local port: 15668
   Foreign host: 10.0.0.1, Foreign port: 179
   
   Enqueued packets for retransmit: 0, input: 0  mis-ordered: 0 (0 bytes)
   
   Event Timers (current time is 0x3A46EB54):
   Timer          Starts    Wakeups            Next
   Retrans            18          0             0x0
   TimeWait            0          0             0x0
   AckHold            22          9             0x0
   SendWnd             0          0             0x0
   KeepAlive           0          0             0x0
   GiveUp              0          0             0x0
   PmtuAger            0          0             0x0
   DeadWait            0          0             0x0
   
   iss: 2047376434  snduna: 2047376784  sndnxt: 2047376784     sndwnd:  16035
   irs:  821061364  rcvnxt:  821062116  rcvwnd:      16188  delrcvwnd:    196
   
   SRTT: 279 ms, RTTO: 500 ms, RTV: 221 ms, KRTT: 0 ms
   minRTT: 24 ms, maxRTT: 384 ms, ACK hold: 200 ms
   Flags: higher precedence, nagle
   
   Datagrams (max data segment is 536 bytes):
   Rcvd: 33 (out of order: 0), with data: 22, total data bytes: 751
   Sent: 29 (retransmit: 0, fastretransmit: 0), with data: 17, total data bytes: 349

7. Select #comparison operator.

8. Type /maximum limit\s(\d+)/ in the Rule field.

9. Select %comparison operator.

10. Type 80 in the Max accepted % field.

11. Type /(\d+) accepted prefixes/ in the Compare with field.

Analytics Templates

Analytic Templates are pairs of commands that originate in Command Templates. The command pairs are shown as Pre and Post on the app interface. Unlike Command Templates, the Analytic Templates do not run commands on devices at execution time, but instead utilizes responses that have been collected at earlier workflow stages by the Command Templates. The output of the Pre and Post commands in the Analytic Templates is evaluated against each other according to a set of associated analytic rules.

Creating Analytics Template

To create a new Analytics Template:

1. Click New Analytic Template. The Analytic Template Designer page opens.

2. Enter a template name and click SAVE (required).

3. Select the global pass flag (All/One commands must pass).

4. Enter the device name in the search field (upper right) to select a device.

5. Select the appropriate Pre Template and Post Template from the dropdown list.

6. Select the appropriate command pair (Pre and Post Command) to test against the device.

7. Click Add Command Pair to add a new command pair.

8. Select the appropriate rule to apply to the command pair.

9. Click Add Rule to add a new rule to a command.

10. Select the Severity.

11. Click Test Run. After evaluating the response, the associated rules will be marked with either a green check mark (passed) or red minus sign (failed). The device response will appear at the bottom of the screen.

    

12. Click Delete to remove a saved Analytic Template. To clone a saved Analytic Template, edit the template name and click Update.

    

Defining Analytics Template Rules

The following rules apply.

Rule	Description
regex	Allows you to extract specific values (using parentheses) and compare them against each other with one of the following relationships. See below for operators.
table	Allows the extraction of key and values from the table dataset, and then compares if all keys and values are the same. It is useful for datasets that may change their order.
matches	Compares the Pre and Post responses entirely. This rule passes when both responses are equivalent. Under the ignore pattern, it is possible (not mandatory) to specify a regular expression to be ignored during comparison (time stamp in the use case below).
!matches	Compares the Pre and Post responses entirely. This rule passes when the responses are different. Under the ignore pattern, it is possible (not mandatory) to specify a regular expression to be ignored during comparison.

Regex Operators

Operator	Description
=	String. Numbers are equal.
!=	String . Numbers are not equal.
<=	Pre-number is less than or equal to Post-number.
<	Pre-number is less than Post-number.
>=	Pre-number is greater than or equal to Post-number.
>	Pre-number is greater than Post-number.
%	Evaluates a percentage value calculation against a threshold. The percentage value is calculated by multiplying 100 times the Post-Regex value divided by the Pre-Regex value: 100 x (Post-Regex / Pre-Regex). The threshold is a static number typed in the Post/Pre % field. The rule returns true if the calculated percentage value is greater than or equal to the threshold; otherwise it returns false.